-
Notifications
You must be signed in to change notification settings - Fork 7.2k
Introducing CVCUDA Backend #9259
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9259
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 49ca2f9 with merge base 9e37c3a ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@AntoineSimoulin has exported this pull request. If you are a Meta employee, you can view the originating Diff in D85862362. |
|
NOTES:
|
|
@AntoineSimoulin has imported this pull request. If you are a Meta employee, you can view this in D85862362. |
Summary:
Users have to explicitly opt-in for those transforms. Here we provide the first building block for this interface. We add the functionals `to_nvcv_image` and `nvcv_to_tensor` to transform `torch.Tensor` to `nvcv.Tensor`. We also implement the corresponding class transforms `ToNVCVImage` and `NVCVToTensor`.
## How to use
```python
from PIL import Image
import torchvision.transforms.v2.functional as F
orig_img = Image.open("leaning_tower.jpg")
img_tensor = F.pil_to_tensor(orig_img)
nvcv_tensor = F.to_nvcv_tensor(img_tensor.cuda())
img_tensor = F.nvcv_to_tensor(nvcv_tensor)
```
> [!NOTE]
> NVCV tensors are automatically converted to NHWC format. Contrary to torchvision convention, which relies on NCHW format.
## Run unit tests
```bash
pytest test/test_cvcuda.py
...
37 passed in 0.15s
```
Test Plan:
```python
from torchvision import _is_cvcuda_available
_is_cvcuda_available()
```
## Run tests
```bash
buck test fbcode//mode/opt fbcode//pytorch/vision/test:torchvision_cvcuda
...
Tests finished: Pass 38. Fail 0. Fatal 0. Skip 0. Build failure 0
```
Differential Revision: D85862362
Pulled By: AntoineSimoulin
e96659c to
c4a50bb
Compare
Summary:
Users have to explicitly opt-in for those transforms. Here we provide the first building block for this interface. We add the functionals `to_nvcv_image` and `nvcv_to_tensor` to transform `torch.Tensor` to `nvcv.Tensor`. We also implement the corresponding class transforms `ToNVCVImage` and `NVCVToTensor`.
## How to use
```python
from PIL import Image
import torchvision.transforms.v2.functional as F
orig_img = Image.open("leaning_tower.jpg")
img_tensor = F.pil_to_tensor(orig_img)
nvcv_tensor = F.to_nvcv_tensor(img_tensor.cuda())
img_tensor = F.nvcv_to_tensor(nvcv_tensor)
```
> [!NOTE]
> NVCV tensors are automatically converted to NHWC format. Contrary to torchvision convention, which relies on NCHW format.
## Run unit tests
```bash
pytest test/test_cvcuda.py
...
37 passed in 0.15s
```
Test Plan:
```python
from torchvision import _is_cvcuda_available
_is_cvcuda_available()
```
## Run tests
```bash
buck test fbcode//mode/opt fbcode//pytorch/vision/test:torchvision_cvcuda
...
Tests finished: Pass 38. Fail 0. Fatal 0. Skip 0. Build failure 0
```
Differential Revision: D85862362
Pulled By: AntoineSimoulin
7e751d3 to
eea836c
Compare
NicolasHug
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for working on this @AntoineSimoulin ! This looks great, I made some suggestions about potentially simplify the features we need to support right now, and about potentially preserving the existing file structure. Let's chat more
|
This looks like a great base implementation, thanks for getting this started @AntoineSimoulin ! My immediate comment would be that we should only be using the cvcuda import. Since cvcuda has been published on PyPi, all of the nvcv objects are aliased inside the cvcuda module. In the 0.16.0 release (to be released end of week) we have removed the nvcv Python module completely to simplify the library. All currently published verisons of PyPi will work using only the cvcuda Python module. This will also reduce any naming convention confusion between NVCV/CV-CUDA tensors which may come up for users. |
test/test_transforms_v2.py
Outdated
| F.to_cvcuda_tensor(img_data) | ||
|
|
||
| @pytest.mark.parametrize("num_channels", [1, 3]) | ||
| @pytest.mark.parametrize("dtype", [torch.uint8, torch.float32, torch.float64]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should the int16/int32 dtypes also be tested in the round trip tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to be sure regarding RGB, CVCUDA only supports torch.uint8 (cvcuda.Format.RGB8) and torch.float32 (cvcuda.Format.RGBf32)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, great question! The formats are actually relevant for the cvcuda.Image datatype and tensors do not have this format info. CV-CUDA tensors should be able to read most datatypes from torch directly. I saw you posted an initial note asking about the difference between image/tensor, and I will post an explanation on this.
For tensors, most operators should support either unsigned int, signed int, floats, or some combination of those.
Would it be helpful at this stage to have a conclusive list of the datatypes/channels CV-CUDA operators can utilize?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gotcha! I think at this point we should remove the _validate_cvcuda_dtype function since cvcuda should support all data types supported by torchvision (torch.uint8, torch.int16, torch.int32, torch.float32, torch.float64). Maybe just reject torch.float16?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the following PRs, yes I think it would be useful to have a list of the datatypes/channels CV-CUDA operators can utilize!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rejecting torch.float16 sounds like a good solution, we are tracking support for fp16 but no timeline as of yet.
Summary:
Summary
-------
This PR provides the first building blocks for CV-CUDA integration in torchvision. We add the functionals `to_cvcuda_tensor` and `cvcuda_to_tensor` to transform from `torch.Tensor` to `cvcuda.Tensor` and back. We also implement the corresponding class transforms `ToCVCUDATensor` and `CVCUDAToTensor`.
**Key features:**
* **3-channel RGB support only**: Simplified API focusing on the most common use case (RGB images)
* **Supported data types**: `torch.uint8` (RGB8 format) and `torch.float32` (RGBf32 format)
* **Lossless round-trip conversions**: Exact data preservation when converting PyTorch ↔ CV-CUDA
* **Informative error messages**: Helpful installation instructions when CV-CUDA is not available
* **Batch-aware**: Handles both unbatched (CHW) and batched (NCHW) tensors
Users must explicitly opt-in for these transforms, which require CV-CUDA to be installed.
How to use
----------
```python
from PIL import Image
import torchvision.transforms.v2.functional as F
# Load and convert image to PyTorch tensor
orig_img = Image.open("leaning_tower.jpg")
img_tensor = F.pil_to_tensor(orig_img)
# Convert to CV-CUDA tensor (must be 3-channel RGB on CUDA)
cvcuda_tensor = F.to_cvcuda_tensor(img_tensor.cuda())
# Convert back to PyTorch tensor
img_tensor = F.cvcuda_to_tensor(cvcuda_tensor)
```
> [!NOTE]
>
> * NVCV tensors are automatically converted to NHWC layout, contrary to torchvision's NCHW default
> * Only 3-channel RGB images and 1-channel grayscale are supported for now
> * Input tensors will be uploaded to CUDA device when converting to CV-CUDA tensors
> * CV-CUDA must be installed: `pip install cvcuda-cu12` (CUDA 12) or `pip install cvcuda-cu11` (CUDA 11)
Run unit tests
--------------
## Run unit tests
```bash
pytest test/test_transforms_v2.py -k "cvcuda"
...
35 passed, 4 skipped, 9774 deselected in 1.12s
```
Test Plan:
```python
from torchvision import _is_cvcuda_available
_is_cvcuda_available()
```
## Run tests
```bash
buck test fbcode//mode/opt fbcode//pytorch/vision/test:torchvision_transforms_v2
...
Tests finished: Pass 38. Fail 0. Fatal 0. Skip 0. Build failure 0
```
Differential Revision: D85862362
Pulled By: AntoineSimoulin
f673259 to
1dcae5c
Compare
|
The primary purpose for A For the purposes of the CV-CUDA backend I believe we can exclusively use the |
Summary:
Summary
-------
This PR provides the first building blocks for CV-CUDA integration in torchvision. We add the functionals `to_cvcuda_tensor` and `cvcuda_to_tensor` to transform from `torch.Tensor` to `cvcuda.Tensor` and back. We also implement the corresponding class transforms `ToCVCUDATensor` and `CVCUDAToTensor`.
**Key features:**
* **3-channel RGB support only**: Simplified API focusing on the most common use case (RGB images)
* **Supported data types**: `torch.uint8` (RGB8 format) and `torch.float32` (RGBf32 format)
* **Lossless round-trip conversions**: Exact data preservation when converting PyTorch ↔ CV-CUDA
* **Informative error messages**: Helpful installation instructions when CV-CUDA is not available
* **Batch-aware**: Handles both unbatched (CHW) and batched (NCHW) tensors
Users must explicitly opt-in for these transforms, which require CV-CUDA to be installed.
How to use
----------
```python
from PIL import Image
import torchvision.transforms.v2.functional as F
# Load and convert image to PyTorch tensor
orig_img = Image.open("leaning_tower.jpg")
img_tensor = F.pil_to_tensor(orig_img)
# Convert to CV-CUDA tensor (must be 3-channel RGB on CUDA)
cvcuda_tensor = F.to_cvcuda_tensor(img_tensor.cuda())
# Convert back to PyTorch tensor
img_tensor = F.cvcuda_to_tensor(cvcuda_tensor)
```
> [!NOTE]
>
> * NVCV tensors are automatically converted to NHWC layout, contrary to torchvision's NCHW default
> * Only 3-channel RGB images and 1-channel grayscale are supported for now
> * Input tensors will be uploaded to CUDA device when converting to CV-CUDA tensors
> * CV-CUDA must be installed: `pip install cvcuda-cu12` (CUDA 12) or `pip install cvcuda-cu11` (CUDA 11)
Run unit tests
--------------
```bash
pytest test/test_transforms_v2.py -k "cvcuda"
...
35 passed, 4 skipped, 9774 deselected in 1.12s
```
Differential Revision: D85862362
Pulled By: AntoineSimoulin
6d0cb9f to
2de2e98
Compare
test/test_transforms_v2.py
Outdated
| img_data = img_data.cuda() | ||
| F.to_cvcuda_tensor(img_data) | ||
|
|
||
| @pytest.mark.parametrize("num_channels", [1, 3]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CV-CUDA can support 2/4-channel tensors, I missed this previously when discussing the validate format function. Same comment for test_round_trip_batched.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good point! Right now torchvision focuses on 1 or 3 channel tensors so I propose we stay align with what is explicitly supported even though CV-CUDA supports more formats.
| Args: | ||
| pic (torch.Tensor): Image to be converted to cvcuda.Tensor. | ||
| Tensor can be in CHW format (unbatched) or NCHW format (batched). | ||
| Only 1-channel and 3-channel images are supported. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should update this comment to include 2/4 channel images since we no longer filter those out with the validation function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@justincdavis thanks for the comments. See my answer above!
|
@justincdavis for |
|
@AntoineSimoulin I have not encountered this previously, as far as I am aware |
|
Summary of the changes:
|
|
@justincdavis thanks for the confirmation, just wanted to double check and indeed I confirmed |
|
Other than the final comment I made, this looks great to me! Thank you @AntoineSimoulin ! |
NicolasHug
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the great work @AntoineSimoulin
and thank you so much @justincdavis for the reviews!
Summary: Summary ------- This PR provides the first building blocks for CV-CUDA integration in torchvision. We add the functionals `to_cvcuda_tensor` and `cvcuda_to_tensor` to transform from `torch.Tensor` to `cvcuda.Tensor` and back. We also implement the corresponding class transforms `ToCVCUDATensor` and `CVCUDAToTensor`. These transforms require CV-CUDA to be installed. How to use ---------- ```python from PIL import Image import torchvision.transforms.v2.functional as F # Create rand ``torch.Tensor`` image (must be 3-channel RGB/Gray and have batch dimension) img_tensor = torch.randint(0, 256, (1, 3, 320, 240), dtype=torch.uint8) # Convert to ``cvcuda.Tensor`` (will be uploaded to CUDA) cvcuda_tensor = F.to_cvcuda_tensor(img_tensor) # Convert back to ``torch.Tensor`` img_tensor = F.cvcuda_to_tensor(cvcuda_tensor) ``` > [!NOTE] > > * ``cvcuda.Tensor`` are automatically converted to NHWC shape (since most CV-CUDA transforms only support this shape) > * Only 3-channel RGB images and 1-channel grayscale are supported for now > * Input tensors will be uploaded to CUDA device when converting to CV-CUDA tensors > * CV-CUDA must be installed: `pip install cvcuda-cu12` (CUDA 12) or `pip install cvcuda-cu11` (CUDA 11) Run unit tests -------------- ```bash pytest test/test_transforms_v2.py -k "cvcuda" ... 338 passed, 9774 deselected in 1.65s ``` Differential Revision: D85862362 Pulled By: AntoineSimoulin
bc5dde8 to
49ca2f9
Compare
Summary
This PR provides the first building blocks for CV-CUDA integration in torchvision. We add the functionals
to_cvcuda_tensorandcvcuda_to_tensorto transform fromtorch.Tensortocvcuda.Tensorand back. We also implement the corresponding class transformsToCVCUDATensorandCVCUDAToTensor. These transforms require CV-CUDA to be installed.How to use
Note
cvcuda.Tensorare automatically converted to NHWC shape (since most CV-CUDA transforms only support this shape)pip install cvcuda-cu12(CUDA 12) orpip install cvcuda-cu11(CUDA 11)Run unit tests
Differential Revision: D85862362